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ABSTRACT 



A method and system for employing image recognition 
techniques to produce a photocollage from a plurality of 
images wherein the system obtains a digital record for each 
of the plurality of images, assigns each of the digital records 
a unique identifier and stories the digital records in a 
database; automatically sorts the digital records using al 
least one date type to categorize each of the digital records 
according at least one predetermined criteria; and generates 
a first draft of a photocollage from the digital records. The 
method and system provides for alterations of the first draft 
using either a sticker selected from a plurality of stickers 
which each have a unique identifier within a data base or a 
digital file format that allows drag and drop features of both 
the photocollage and alternative images to be placed within 
the photocollage. The system employs data types selected 
from pixel data; metadata; product order information; pro- 
cessing goal information; or customer profile to automati- 
cally sort data typically by culHng or grouping to categorize 
according to either an event, a person, or chronologically. 

18 Claims, 9 Drawing Sheets 
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FIG. 1 




06/30/2003, 



EAST Version: 



1.03.0002 



U.S. Patent May 28, 2002 Sheet 2 of 9 US 6,396,963 B2 



FIG. 2 
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FIG. 3 
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Fig. 5 (Analog version) 
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Fig. 6 (Digital version) 
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PHOTOCOLLAGE GENERATION AND 
MODIFICATION 

CROSS-REFERENCE TO RELATED 
APPLICATIONS 

This application is related to formerly filed applications 
filed by the same inventors and assigned commonly with the 
present application for invention entitled "PhotocoUage 
Generation And Modification Using Image Recognition" 
filed on Nov. 25, 1998; and "System And Method Of 
Constructing A Photoalbimi" filed on Aug. 19, 1998. 

HELD OF THE INVENTION 

The invention relates generally to the field of 
photography, and in particular to photo collections. More 
specifically, the invention relates to generating photocol- 
lages automatically and providing for alteration of the 
automatically created photocollage. 

BACKGROUND OF THE INVENTION 

Photographs, videos, and memorabilia collections are 
very commonly used to maintain memories and events that 
form part of an individual's life. These collections serve to 
augment the human memory and enrich the process of 
sharing stories related to the memories. When organized, 
viewed and shared on a regular basis a collection of memory 
artifacts generates a large reward, enriching the lives of all 
involved. The nature of these collections is such that they 
grow steadily, event by event, year by year, and soon 
become large and difficult to manage. Collections of photos 
and memorabilia are considered one of the most important 
and valued possessions by most people. They are the first 
things that people think of when forced to flee their homes 
due to fire, flood or other natural disaster. These collections 
possess intrinsic, emotional value, even if they are never 
viewed, because the need to preserve a memory of life is 
strong and universal. Because of the relative importance of 
these memories to the persons involved, the prior art is 
replete with teachings that disclose organizational methods. 

The most common manner of organizing these collections 
within the prior art is to place the photos, videos or memo- 
rabilia into either an album or a box. Common vinyl album 
pages provide the means to store and view between one and 
five standard sized photos per page. Creative people often 
spend hours carefully selecting and arranging photos, writ- 
ing captions, clipping newspaper articles, and other memo- 
rabilia to create visual stories or scrapbooks. Once organized 
into groups or pages these photocollages greatly enhance a 
person's ability to remember and share the story surrounding 
the depicted events. These simple organization tools allow 
the collections to be easily viewed and also serves to protect 
the artifacts themselves. There are numerous types of 
albums and boxes available in the market today, ranging 
from simple vinyl sleeves to boxes manufactured from 
specialized materials designed to preserve the artifacts. 
Album vendors include Pioneer Photo Albums, Design 
Vinyl and Cason-Talens. Box vendors include Exposures, 
None of these prior art disclosures provide a means by which 
a photocollage of these memorable events can be easily 
constructed by persons to who these event means so much. 

As used herein photocollage refers to a single page having 
a plurality of images, such as a page in a photo album, or a 
composite image having a number of images relating to a 
single theme such as a vacation, wedding, birthday party or 
the like. The concept of photocollage as used herein also 



1,396,963 B2 

2 

includes the concept of a bound photo album having a 
plurality of pages, one or more of which is a photocollage. 

Despite the fact that many people are engaged in collect- 
ing these memorable artifacts, few people have the free time 
5 available to invest on a regular basis to organize and 
maintain them. Before long, the amount of unorganized 
material becomes a significant psychological barrier to get- 
ting organized. Other barriers exist which prevent people 
from actively maintaining these memorabilia collections 
10 such as confidence in their process, access to the materials, 
or remembering the details about the event. Often, once 
people get started on this organizational task they find it 
rewarding and fun, but stiU a significant amount of work. 
Many attempts have been made to provide tools for 
15 working with or organizing photo and memorabilia collec- 
tions. Computer software programs such as Picture-It™, by 
Microsoft, or Creative Photo Albums™, by Dog Byte 
Development, allow people to work with digital versions of 
their photos and create digital versions of an album or print 
20 them on a home printer. Software products such as these 
require each photo or artifact exist in digital form before 
they can be used. Although these products increase the 
ability to change and enhance photos and scanned memo- 
rabilia they do not reduce the amount of work needed to 
25 organize collections or create visual stories. Other services 
such as Photo -Net™ by Picture Vision™ will scan photo- 
graphs in a high-quality format at the time of photo pro- 
cessing and provide a thumbnail image of the scanned 
images via the Internet. A customer, using these scanned 
30 images can create collections of photos which can be viewed 
on the Internet or have prints generated. Currentiy some of 
these services do not allow for the arrangement of several 
photos on a page and are limited to consumers who have a 
collection of digital images and a computer connected to the 
35 Internet and who are both computer and web literate. 

It should be apparent from the foregoing discussion that 
there remains a need within the art for a method by which 
consumers can create photocollages and photo albums (or 
have them made for them) in a manner that is as simple as 
40 ordering prints. 

SUMMARY OF THE INVENTION 

The present invention addresses the shortcomings within 
the prior art and provides an improved method of generating 
45 photo albums from consumer photographs that requires a 
minimum amount of efibrt but yields a high-quality product 
and is reasonably priced. 

The present invention is directed to overcoming one or 
more of the problems set forth above. Briefly summarized, 
50 according to one aspect of the present invention, a system 
and method for producing a photocollage from a plurality of 
images, comprising the steps of: a) obtaining a digital record 
for each of the plurality of images, each of the digital records 
having a unique identifier and storing the digital records in 
a database; b) automatically sorting the digital records using 
at least one dale type to categorize each of the digital records 
according at least one predetermined criteria; c) employing 
means responsive to the sorting step to compose a photo- 
collage from the digital records. The system then associates 
60 each of the images with at least one of the categories 
followed by a sorting step that arranges the images accord- 
ing to the categories. The system then employs the catego- 
ries to automatically construct the photocollage from the 
stored images by generating a plurality of pages of the stored 
images. 

These and other aspects, objects, features and advantages 
of the present invention will be more clearly understood and 
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appreciated from a review of the following detailed descrip- envisions providing sufiBcient image understanding of the 

tion of the preferred embodiments and appended claims, and subject images with the ultimate goal of teaching the system 

by reference to the accompanying drawings. to understand that there arc pervasive themes that exist 

within various sets of images and recognizing these themes 

ADVANTAGEOUS EFFECT OF THE 5 thereby creating an image story product, or photocollage. 

INVENTION As shown in FIG- 2, the collection step begins when a 

rn. . ■ 1. *u 11 ■ J- . customer, having completed picture taking for one or more 

The present mvention has the followmg advantages: , j \- . . - 

Allows the user to have (1) an easy method for creating Z'""^- ''"'1^7,°''" °' ^"^"''^ °u ^''^ f 

professional looking photocollages, (2) duplication of 1"' ''•f»l °»^'°°'y '^"ds 12. photographic 

L . n A /%\ \ if* 11 /i c 1 . 10 pnnts 6 or Video camera media 8 to a processing facility 14. 

photocollages, and (3) keeping photocollage files for later a. ,u *u ♦ *u * ^ i- j ci 

& » \ / r & r b At the time that the customer delivers the exposed film 

cartridge(s) to the processing facility the customer's identity 

BRIEF DESCRIPTION OF THE DRAWINGS ^ recorded and associated with the suite of film cartridges 

and other image sources. Alternatively the customer identity 

FIG. 1 is a block diagram illustrating the basic system 15 may be encoded on the film by exposing a machine-readable 

elements used in practicing the present invention; and sequence of marks along the edge of the film or by using the 

FIG. 2 is a system diagram showing the collection steps magnetic coating on the Advanced Photo System film. The 

that take place once a customer has delivered images to the conventional tilm processing takes place with conventional 

system; correction of exposed photographs to balance color and 

FIG. 3 is a flow diagram showing the active processing 20 brightness and contrast. The exposed film images are chemi- 

goal steps used by the system of the present invention; ^^^^y processed 16 to generate an optical image. These 

FIG. 4 is a flow chart showing the steps performed by the |. '"^f"' '^f''^ ^^"^^^ . *^ Pf^/^^^ ^ jl^^^ 

present invention towards a story preparation based photo- '"^"^"^^ ^^'f 'J^'} ^ ' ^J"'' 

^ r r r general, to produce a high-resoluUon printed image of 8x10 

' 25 inch size, a resolution of 1538x1024 pixels is required. 

FIG. 5 IS a flow chart showing the steps performed by the digital still Camera images from Digital Still Camera 

present mvenUon towards preparation of a photocollage that , ^^^^,y ^ards that are delivered to the processing facility 

allows alterations after automatic creation; ^re digitally processed 22 to produce an equivalent digital 

FIG. 6 is a flow chart showing the steps performed by the image file of similar size and stored on the data store 20. 

present invention; towards preparation of a photocollage that 30 Analog video camera images on media 8 delivered to the 

is in a digital file format that can be sent across the internet; processing facility are digitized and processed before stor- 

FIG. 7 is a flow chart showing the steps performed in the on the data store 20. At the conclusion of this step in the 

processing facility illustrated in FIG. 2; process there exists a collection of digital image files asso- 

HG. 8 is a schematic diagram showing a sUcker sheet of ^^^^^^ ^ customer by means of a customer identifica- 

reduced resolution images to be used with the present ^5 tion. This collection of digital image data, or pixel data 32, 

invention; ^ available as a source of data for the Story Services 

a ' J • u- r*Li processing modules. Data attached or associated with indi- 

MG. 9 is a drawing showing some of the layout styles -j 1 ■ r ■ j j u .1, 

available for a page in a photOK^llage. vidual images or groups of images provided by ttie customer 

^ ^ ^ ^ such as date, Ume, location, sound, camera identification, 

To facilitate understanding, identical reference numerals ^ customer identification, exposure parameters. Advanced 

have been used, where possible, to designate identical pho^o System IX data, are all examples of meta-data. This 

elements that are common to the figures. meta-data 34 is stored on image store 20 and associated with 

DETAILED DESCRIPTION OF THE available pixel data 32. In addition the customer selects 

INVENTION desired output product or product type from a listing of 

45 available choices. This selection may be indicated by the 

An acceptable photo album or photocollage can be ere- customer marking the photo processing bag 24, verbally 

ated automatically from customers' exposed film, negatives, indicating the desired product to a clerk 26 or by input at a 

or digital images. computer terminal or kiosk 28. The order description 36 

Referring to FIG. 1, which is the method as envisioned by generated from any one of the indicated sources provided by 
the present invention designed to automatically produce 50 the customer is stored on the system and is associated, by 
photocollages including albums 17, CDs 15 and other means of a customer identification, with an existing cus- 
image-based keepsakes 11. The process of transforming the tomer profile 30 in the data store 20. 
supplied image material into a photocollage is referred to as At the conclusion of the collection step the pixel data, 
Story Preparation services. Story Preparation services meta-data, customer profile and order description are avail- 
applies the necessary amount of image understanding 55 able as input data for the subsequent processing steps. These 
embodied in a collection of processing modules executed in steps can be performed in the order listed or they may be 
a non-linear sequence to create a story representation of the rearranged in different sequences depending upon the 
images. In a preferred embodiment these processing mod- desired output product. The production of a particular pho- 
ules include: collecting, sorting, culling, annotating, tocollage product may include one, two or, more of the 
grouping, enhancing, associating and, composing. These 60 available steps. Which steps are used and in what sequence 
steps can be performed in the order listed or they may be is determined by the system based upon the desired product, 
rearranged in different sequences depending upon the Following the collection step but before subsequent pro- 
desired output product. The production of a particular pho- cessing steps are performed, the specific processing to be 
tocollage product may include one, two or, more of the applied to the collected set of image material is determined 
available steps. Which steps are used and in what sequence 65 in a process goal generation step 37 and embodied in a set 
is determined by the system based upon the desired product. of photocollage processing goals 38. For each of the output 
It is important to remember that the present invention products requested the system determines an optimum 
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sequence of processing steps to be applied. The processing the customer profile is updated with every order processed 
of the output product or products will follow this system to reflect changes in order preferences, order history and 
determined processing sequence, unless the sequence is update uncertain information. In order to supply the system 
modified by one of the processing modules. The processing with the necessary amount of image understanding required 
output for each module is stated as one or more processing 5 to arrange the images into set in accordance with themes 
step goals. The processing in each step uses the stated relating to predetermined criteria that must be provided for 
processing step goals to control the logic, rules, and param- the system to have the capability to identify attributes within 
eters applied during the processing of the collected image the images. For example, in a preferred embodiment, if a 
material to satisfy the stated processing step goal or goals. customer requests a birthday photocollage the system will 
In addition to accomplishing processing goals each process- 10 retrieve the default processing goals which indicate thai the 
ing step can also create goals for itself or future processing steps of collecting, sorting, culling, annotating, and corn- 
steps. New goals, which result from executing a particular posing will be involved in the processing of the requested 
processing step, reflect necessary adjustments of desired product. In addition the processing goals for each of the 
parameters because the original goal is impossible to modules will reflect the default attributes necessary to 
achieve or the processing step may generate new goals for 15 process a birthday product. In this example the sorting 
subsequent processing steps. The initial set of processing module processing goals will include a sort by date goal and 
goals 38 is determined by retrieving a default processing a sort by content goal. The sort by date processing goal is 
description from a database of available products 40 main- further refined to sort the images that occur on or near a 
tained by the system . Once the initial set of processing goals target list of dates which are determined from a list of loiown 
is determined the photocollage processing is initiated by the 20 birthdays retrieved from the customer profile into the pho- 
system. tocollage. 

As described, the automatic processing of photocoUages In a preferred embodiment each processing module per- 
is carried out in processing modules employed to perform forms processing on two distinct levels: objective and sub- 
these image understanding and interpretation steps. Each jective. The objective processing deals with factual infor- 
processing module relies on data from a number of sources 25 mation about the images or groups of images or with data 
in order to satisfy the processing goal. Each processing calculated deterministic algorithms. Examples of factual 
module has access to five sources of data: pixel information data include the size of the image, capture parameters, 
contained in the individual images, meta-data attached to histograms, image transforms, etc. Commercially available 
images or groups of images, the original product or service software programs such as Adobe Photoshop® and Corel 
order description, the list of active processing goals main- 30 Draw® process images using objective data. Subjective 
tained by the system and, a customer profile containing processing deals with information that is uncertain or data is 
information about the customer. The pixel data 32 for each the result of non-deterministic algorithms. Often subjective 
image provides each module with the opportunity to apply results carry with them a confidence factor which allows 
image processing and image understanding algorithms. The subsequent processing steps to interpret the results. Usually 
meta-data 34 as used herein refers to the information 35 subjective results occur when attempting to determine 
attached to the individual images and to groups of images abstract information about an image or group of images, 
will contain information about the image or group of images Examples of this type of the processing would be face 
which originated at the time of capture or was generated in detection, face recognition, facial expression determination, 
prior processing of the image or group of images. Date, time, location determination, assertions, interpretations, etc. Com- 
location, sound, camera identification, customer 40 mercially available software programs such as FacelT© by 
identification, exposure parameters, Advanced Photo Sys- Visionics Corp. process images to associate faces in images 
tern IX data, are all examples of meta-data that are, in a with names and other information. Some processing mod- 
preferred embodiment, be attached to the original input ules process only subjective information, others process 
images. The original product or service order description 36 only objective information and stifl others process both, 
contains the specific product request from the customer and 45 As shown in FIG. 3 the processing of a photocollage is 
any notes that were captured in the ordering process. At any directed by the system using the active processing goals 38, 
time in the processing of the image or group of images the 48, and 49. Because the processing goals are non- 
system will have a list of active processing goals 38. Active deterministic, vary by requested product and may be modi- 
goals are the processing goals for each processing module, fied during the processing there exist a large number of 
which have yet to be accomplished by the system. At the so possible processing sequences. The progression of process- 
start of processing this list will include the specific product ing applied to the collected image material is apphed in 
or products requested by the customer, along with the sequential steps. Each step implements a single processing 
translation of these product goals into system and module module that is intended to satisfy or partially satisfy an 
goals. The customer profile includes both factual and uncer- active processing goal. At each step in the process any one 
tain information related to the customer. The factual data 55 of the available processing modules (culling 62, grouping 
would include general items such as name, address, names 64, enhancing 66, annotating 68, associating 70, or com- 
and ages of individuals in the household, important dates, posing 72) may be executed. At the conclusion of the 
anniversaries, product preferences and purchase history. In processing of a particular step the active processing goals 38 
addition factual data such as face recognition feature vectors that existed before the step processing are updated to reflect 
of the immediate family and extended family, voice training 60 the changes in goals that resulted from the execution of the 
sets, handwriting samples, would also be included in the processing step 52. These updated active processing goals 
customer profile database. Uncertain information would 48 serve as the input processing goals for the subsequent 
include processing goals recommended by processing mod- processing step 53. Each of the available processing mod- 
ules from previous order processing, image understanding ules will be described. In addition to the processing goals 38, 
assertions about the contents of the image or groups of 65 48, and 49, the processing modules at each processing step 
images which have not been verified or other unverified have access to four sources of information: pixel data 32 
information or assertions. Information and data contained in contained in the individual images, metadata 34 attached to 
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images or groups of images, the original product or service images according to the preferred organization scheme of 

order description 36 and, a customer profile 30 containing the customer. The attributes of typical organization schemes 

information about the customer. This process can iterate as would include organizing by events such as birthday party, 

many times as necessary to complete the desired product vacation, holiday, graduation, organizing by time, and orga- 

objectives. 5 nizing by people. Customer requested products 36 could 

The processing of the collection of digital image material also dictate grouping parameters in accordance with the 
is reviewed to remove unwanted images. This step is called product design. Examples of these grouping schemes 
culling. CuUing is performed on images that are deemed include grouping by location in a theme park, by common 
unwanted due to poor quality, or if there are several images life-cycle event such as graduation, by image content type 
that are similar in composition and only one is desired. To lo such as outdoor, indoor, nature, person, group, etc. A variety 
accomplish this culling process involves the computation of of software algorithms exist which are applicable to group- 
several image based metrics using the pixel data 32. These ing. One class of software algorithms operates on metadata 
metrics fall into two basic categories: technical quality 34 associated with the image where the grouping process is 
mecisures and abstract feature metrics. The technical quality similar to searching a media database. Several commercial 
measures would involve calculations to assess the overall 15 examples of media databases that allow searching are Pic- 
sharpness of the image, exposure quality, grain quality of the ture Network Incorporated, Publishers Depot 
image. Algorithms used to calculate technical quality mea- (www.publishersdepot.com). A second class of algorithms 
sures are common in the art, examples would include: employed for this grouping process would include image 
"Estimation of Noise in Images: An Evaluation" by S. I. processing algorithms which identify objects and feature 
Olson in, 55 (4), 1993, pp. 3 19-323 and "Refined filtering of 20 sets within an image by way of the pixel information 32. 
image noise using local statistics" by J. S. Lee in Computer These objects could represent typical cultural icons such as 
Vision Graphics and Image Processing, 15, 1981, pp. birthday cakes, Christmas trees, graduation caps, wedding 
380-389. dresses, etc. An example of such algorithms is reported by 

Images are then given an overall rating of technical J- Edwards and H. Murase, Appearance Matching of 
quality based on the technical quality metrics. The overall 25 Occluded Objects Using Course-lo-fine Adaptive Masks" in 
image quality metric is then compared to a threshold asso- Proceedings of the IEEE Computer Society Conference on 
ciated with the processing goal. Images whose quahty rating Computer Vision and Pattern Recognition, June 1997, pp. 
falls below the threshold are flagged as unwanted. The 533-546. The Product Order Information 36 can also be used 
specific threshold applied to a given image or set of images in this grouping by simply stating the event or location or 
is parametrically determined using the processing goal 38 30 person on which to base the final product. The Processing 
and the customer profile 30 as inputs. In addition to the Goals 38 can also be used as a grouping tool where the 
determination of specific objective image features, abstract specific products offered mandate a specific grouping, 
subjective image features are also calculated. These subjec- The annotation step seeks to generate annotation that can 
live measures of the image allow the module to cull be composed on the story product or photocollage. Anno- 
unwanted or unnecessary images. A typical example of 35 tation is designed to provide context information about the 
images which require culhng occurs when several images in image or group of images and assist the customer in com- 
the group are similarly composed and contain nearly iden- municating the story surrounding the images. A very com- 
tical content such as a group photo which is repeated several mon form of annotation is text associated with each image 
times. In order to choose the best of this group of images the and with a group of images which explains the "who, what, 
digital image is analyzed to determine a quality metric 40 when, where, and why". Such context information is gen- 
composed of both objective and subjective quahty features. erally derived from metadata 34 generated in previous 
The basis for this subjective assessment would be the processing steps, user profiles 30 or, image understanding 
presence or absence of faces, the identification of the faces algorithms applied in the annotation module. "Who" infor- 
in the image, the number of faces present, if the eyes are mation may be determined using face recognition algorithms 
open on each face, if the subject is smiling, or orientation of 45 applied to the pixel data 32 tuned by the training data 
the face to the camera. Examples of algorithms used to contained in the customer profile 30. Using a commercial 
determine subjective features of an image are described in product such as "Facelt" by Visionics, a database of known 
Proceedings of the IEEE Computer Society conference on faces can be retrieved from the customer profile 30 and used 
Computer Vision and Pattern Recognition, June 1997. These to guide the face recognition software. "What" information 
subjective features provide clues about the subjective quality 50 can be asserted by correlating date and time of the image 
of the image. The objective and subjective features calcu- capture available in the image metadata 34 with key dates 
lated for each image are combined and the images are ranked contained in the customer profile 30. In addition more 
according to the relative importance of each feature ana- "what" data can also be asserted by looking within the image 
lyzed to the processing goal 38 and which matches the stored 32 for specific cultural icons, derived from object recogni- 
customer preferences 30. The ranking, because it is partially 55 tion algorithms, such as birthday cakes, Christmas trees or, 
based upon subjective information also carries a probability graduation robes. "When" information for annotation is 
weighting factor. The combination of the ranking and the easily determined using metadata 34 such as date and time 
probability weighting factor is used to assert a decision recorded at the time of capture. "Where" information may be 
about the quality of the image. This quahty assertion is provided from the capture device integrated with GPS 
retained with the image as metadata 34 and is passed to the 60 (Global Positioning System) technology and then added to 
next processing step. In the grouping step images are the metadata 34 for the image or it can be guessed at using 
grouped according to criteria derived from the analysis of image understanding algorithms which correlate known 
the customer profile 30, the active processing goals 38 and location scenes with the image content from available 
the requested product or service 36. The goal of grouping images 32. "Why" information is very difficult to determine 
images is to associate images that are a part of a common 65 without some input from the image owner via the product 
theme or story in the mind of the customer. Typical group- order information 36. The annotation, once determined, can 
ings derived from customer profiles 30 would be to associate be rendered as text, graphics, images, video or sound and 
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associated with the individual images or with the group of 
images in cither a traditional photocollagc or digital product. 

The image enhancement module applies image process- 
ing to improve the image for the intended story based 
purpose. Two categories of image enhancement are consid- 
ered for each image. Basic image enhancements are applied 
to all images, as necessary, would include: red-eye removal, 
brightness and contrast adjustment, color adjustment, and 
crop and zoom to improve composition. Image enhance- 
ments are applied directly to the pixel data 32 and recorded 
in the image metadata 34 from the images. A second 
category of image enhancement is applied to the images in 
order to enhance their use within a story context to com- 
municate the emotion or to support the story theme as 
described in the processing goal 38. For instance, selected 
images from a wedding story would be softened using a blur 
algorithm or the colors of an image could be made to match 
those found in a comic strip. The application of this type of 
enhancement processing is either specified in the product 
description 36 generated from the customer profile 30. All of 
these operations are available in image editing software 
programs such as Adobe PhotoShop, with the exception of 
red-eye removal and red-eye removal can be accomplished 
via the Kodak Imaging Workstation and other software such 
as Picture-It from Microsoft. 

Some product specifications will include a processing 
goal to associate external content that is relevant to the 
product and the image content. This associated content 
provides additional context and interest in the final rendered 
product. External content takes several forms including 
weather reports, newspaper headlines, stock photographs, 
advertisements, historical references, travel brochure copy, 
popular music clips, video segments, etc. In order to locate 
and find appropriate content for the image story/ 
photocoUage the metadata 34 attached to each image and 
image group and is interrogated to derive searchable topics 
or specific information located in the customer profile 30. 
These search topics would focus on the when, where, what 
aspects of the image story. Once a series of potential search 
topics have been assembled they are formatted into queries 
and searches are performed on a variety of databases. The 
results of these content queries is then refined by applying 
priority rules from the product description database and 
customer preferences stored in the customer profile 30. 
These types of searches are common in information data- 
bases such as Yahoo® on the internet (www.yahoo.com) or 
from Dialog corp. If there are items within the image that 
can be identified with specific events (such as a wedding or 
birthday) or if a person can be identified from imaging 
algorithms, the Pixel Information 32 is used to determining 
the types of additional content that are added. The Process- 
ing Goals 38 dictate the forms and types of associated 
content that is added. The Product Order Information 36 is 
also a source of information regarding the association of 
content by a special request on the form (such as a theme 
park or wedding motif). 

The layout processing module places the image data, 
annotations, and associated content into an output ready 
format. This layout step must be repeated for each product 
that was requested by the customer and for each layout goal 
that was generated during the processing steps. The main 
task involved in the layout of a multi-page photocollage is 
the determination of which images are to be placed in 
specific locations on specific pages. In addition the external 
content which has been associated with the individual 
images or groups of images must be composited onto the 
page. The layout process may be accomplished in numerous 
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ways. One means of performing this processing step is to 
employ a parametric photocollage description. This descrip- 
tion specifics the general structure of the photocollagc but is 
not sufficient, of itself, to specify a final photocollage 

5 product. The description file includes photocollage features 
such as number of sections, size and format of individual 
pages, maximum and minimum number of pages allowable, 
maximum and minimima number of customer images allow- 
able per page, section descriptions, cover designs, location 

10 of allowable stock content, etc. By employing a photocol- 
lage description file a variety of tailored photocollage prod- 
ucts may be designed incorporating features specific to the 
design goals. The layout step begins by reading the photo- 
collage description file to initialize a layout algorithm. The 

15 layout algorithm then works page by page and section by 
section to apply specific layout rules guided by the values in 
the photocollage description file. A variety of page/section 
layout algorithms could be employed which embody differ- 
ent design philosophies for photocollage products. The final 

20 photocollage product can be rendered on a variety of output 
media types which would include paper, fabric, as a digital 
file or on any one of a number of digital media forms such 
as a CD. The Processing Goals 38 are used to determine the 
capabilities of the specific devices being used. Customer 

25 Profiles 30 are xised to determine color preferences, layout 
preferences, and design considerations. Metadata 34 is used 
to determine the placement of associated content with spe- 
cific images. In a preferred embodiment, the first cut version 
of the photocollage is rendered as a scaled replica of the final 

30 version. This scaled replica is also identified using a tem- 
plate ID in the form of a bar code, 

FIG. 4 shows the system diagram for Story Preparation 
Services. Input to the system comes from Personal Com- 
puters 90 (using albuming software such as Microsoft Pic- 

35 ture It, Family Base from Micro Dynamics or a myriad of 
others), an interactive kiosk, via the phone or over-the- 
counter order forms 84, a retail oudet 82 with links to the 
system, or digitization services 80 specializing in converting 
analog material into digital information. The information 

40 needed to perform the story services is communicated via 
traditional means including mail, phone, modem, Internet or 
other on-line services. The components required by such as 
system include a digitization system for pictures, video and 
audio 104, a storage facifity 102, an operator interface 96, 

45 algorithms and software 98 for the analysis of the data and 
to provide the necessary steps to complete the product, and 
an output delivery system 100 for printing or other media. 

FIG. 5 is a flow chart showing the steps performed by the 
present invention towards preparation of a photocollage that 

50 allows alterations after automatic creation. The automatic 
creation of a first cut version of a photocollage proceeds as 
described above. After the first cut version is complete, the 
album is reviewed by the customer to determine if any 
changes are desired. In a preferred embodiment the user is 

55 also supplied with a set of scaled image stickers and tem- 
plates to be used to modify the first cut version of the 
photocollage. The sticker images are generated at the time of 
the original photocollage processing as described below. 
These stickers and templates are illustrated and described in 

60 FIGS. 7, 8 and 9 and the discussion related, thereto. A 
customer wishing to construct additional pages for a pho- 
tocollage or to modify the page layout of the first cut version 
selects one or more images by peeling the sticker image 
from the sticker sheet 118 and placing it on the page layout 

65 form 160 as shown in FIG. 9. The page layout form 160 
includes one or more scaled graphic depictions of the final, 
page each comprising a unique page layout. In one embodi- 
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ment each scaled page representation includes one or more 
graphic image boxes each of which is an allowable location 
to place a sticker. Different scale page layout representations 
depict the available album page layouts made available by 
the service. Normally this variety of page layout choices 
would include variety along the dimensions of: the number 
of images per page, the resulting size of the images on the 
final printed page, arrangements accommodating both por- 
trait (vertical) and landscape (horizontal) captured original 
images, different final printed page sizes, provision for 
annotation text, different size original images including 
standard 35 mm, Advanced Photo System classic, HDTV 
and Panoramic images and. Digital Still Camera formats. 
The scale of the sticker images and the scale of the graphic 
image boxes on the page layout order form do not have to 
be the same. If the page layout form image box is larger than 
the sticker image the image is enlarged to fill the box when 
producing the final printed album page. If the page layout 
form image box is smaller than the sticker image the image 
is reduced to fill the box when producing the final printed 
album page. In this way a variety of final output sized 
images can be made available to the user without the need 
to produce multiple sizes of sticker images. The page layout 
sheet and cover sheet would have identifiers 164 on them so 
that the page type can be identified during the scanning 
process. Bar code or alphanumeric characters can be used 
for this identification. 

Other means to alter the placement of images in the 
photocollage can also be used including the use of a marking 
means using an "X" over the image to indicate deletion and 
a circle around the image and an arrow showing where the 
image is being moved on the photocollage. Photocollage 
templates can also be modified by other means such as the 
selection via ID numbers representing layouts in a catalogue 
showing the layout choices. After alteration, the modified 
photocollage is brought or sent to a location, such as a 
retailer, to be scanned and printed in final form. 

f^G. 6 is a flow chart showing the steps performed by the 
present invention; towards preparation of a photocollage that 
is in a digital file format that can be sent across the internet. 
The process is similar to that shown in FIG. 5 except that all 
steps are performed on a digital computer with images being 
transmitted as digital files. It is envisioned that any 
alterations, both for changing the images within a page and 
changing the templates describing the layout of the pages, 
are performed by employing traditional software method- 
ologies like drag-and-drop or choosing from a list for 
replacement of images and templates. In this digital case, the 
information needed to convey the changes can be sent 
electronically to the retail site for printing of the final 
photocollage. 

Scaled image stickers used by the consumer to correct the 
first cut version of the photocollage are generated by using 
the original high-resolution images archived in processing 
facility 14 as seen in FIG. 2. This archive of digital images 
would, in most cases, comprise a software database appli- 
cation and a large collection of optical or magnetic storage 
media. In order to allow for the future association of the 
high-resolution images with their corresponding reduced 
resolution thumbnail images, each image is assigned a 
unique identifier. There are a variety of means, known in the 
art, for the generation of this unique identification. One such 
means is disclosed in Townsend and Shaffer, U.S. Pat. No. 
5,784,461, in which a combination of the customer identi- 
fication information and the photofinishing location, 
equipment, and date and time are combined. Other means of 
unique identification could be employed so long as they 
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provide a one-to-one association of a person to a single 
image. The assigned unique image identification is associ- 
ated with both the high and reduced resolution versions of 
the image. The unique identifier embodiment can be any of 
5 either: embedded data contained within the pixels 
themselves, invisible ink, visible codes, histogram, color 
information, or the like. The reduced resolution images are 
further processed to generate a formatted array or sticker 
sheet 118 as seen in FIG. 8. 

10 Referring to FIG. 8, the sticker sheet 118 also includes an 
alpha numeric human readable image identifier 119 which 
allows the user to locate the corresponding photographic 
print of the sticker sheet image. In addition a unique 
machine-readable image identification is also included in the 

15 printed sticker sheet. The machine readable image identifi- 
cation could take the form of a bar code, machine readable 
characters or any one of a number of data-encoding patterns. 
A preferred implementation this unique machine readable 
image identification is to embed it directly into the pixels of 

20 the thumbnail image using signal processing techniques. An 
example of such techniques can be seen in Daly et al. (U.S. 
patent application Ser. No. 08/565,804). 

The method of embedding digital data in an image can be 
best considered in two stages; an encoding process and a 

25 decoding process. First, a multi-level data image is gener- 
ated from digital data. The multi-level data image as 
described in detail below is an image having a constant 
background value and an array of spots on the background 
representing the digital data. The data image may be pro- 

30 duced using digital image processing techniques, or may be 
produced optically for example by exposing spots on a 
photographic film. Next, the data image is convolved with 
an encoding carrier image to form a frequency dispersed 
data image. The encoding carrier image is preferably a 

35 rotationally symmetric, low amplitude, high frequency pat- 
tern employed to spatially disperse the data image to mask 
its visibility when added to the source image. Preferably the 
convolution is performed on a digital computer using a well 
known Fourier Transform method and digital versions of the 

40 data image and the encoding carrier image. The convolution 
may also be performed using a direct convolution algorithm 
on the digital computer. Alternatively, the convolution may 
be performed optically using well known optical convolu- 
tion techniques and optical versions of the data image and 

45 the encoding carrier image. The frequency dispersed data 
image is then added to the source image to form a source 
image with embedded data. As described in more detail 
below, the addition may be performed either optically using 
traditional photographic processes, or digitally using a digi- 

50 tal computer and digital versions of the source image and the 
frequency dispersed data image. If the addition is performed 
digitally, a hard copy version of the digital source image 
having embedded data may be produced using a digital 
printer such as a thermal, ink-jet, electrophotographic, or 

55 silver halide printer. 

Once the generation of the final photocollage from the 
modified first cut version of the photocollage has been 
accepted or modified by the user the modified version is 
rendered into a final photocollage. These modified photo- 

60 collage orders are scanned to generate a final version pro- 
cessing order. If the modified photocollage contains image 
stickers the photocollage processing identifies the original 
high resolution data by recovering the embedded digital 
identification data. The digital identification data is recov- 

65 ered from the source image having embedded data by first 
cross correlating the source image having embedded data 
with a decoding carrier image to produce a recovered data 
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image. Preferably, the decoding carrier image is identical to 
the encoding carrier image and the cross correlation is 
performed by a digital computer on digital versions of the 
source image having embedded data and the decoding 
carrier image. If a hard copy of the source image having 5 
embedded data is used, the hard copy is first scanned as 
described in detail below to produce a digital version. 
Alternatively, the cross correlation may be performed using 
well known optical techniques and optical versions of the 
source image having embedded data and the decoding 
carrier image. Finally, the digital data is extracted from the 
recovered data image. Preferably, the digital data is 
extracted from a digital version of the recovered data image 
using a pattern recognition process in a digital computer. 
Alternative methods for encoding data within an image can 
be found in issued U.S. Pat. Nos,: 5,636,292; 5,128,525; 
5,285,438; and 5,126,779 which can also be used with the 
present invention. 

Referring again to FIG. 2, after scanning the modified first 
version photocoUage and template forms the processing 20 
facility uses the embedded image identification to generate 
the list of images needed to generate the final photocoUage. 
Using the list of images the processing facility 14 finds and 
retrieves the corresponding high resolution images that were 
previously stored in the image storage 24 which are asso- 25 
ciated with the unique image identification numbers that 
were supplied by the scanned layout sheets 160. These 
high-resolution images are then assembled into a photocol- 
lage using a page description language or template following 
graphical rules. These graphical rules incorporate customer 30 
preferences such as size of the pages, background styles and 
colors, etc. In this page layout process it may be necessary 
to zoom and crop the images to optimize the layout or to 
match customer preferences. 

FIG. 7 illustrates the process steps that are performed 35 
from developing photographs to generation of photocoUages 
by the processing facility of FIG. 2. The customer brings in 
exposed film 142 or a memory card from a digital camera 
143 containing images that are to be made into a photocol- 
lage. The photofinisher generates traditional prints 146 for 40 
the customer. In the case of film, the fihn is scanned 145 to 
create a digital image. Once acquired, the digital images are 
assigned a unique ID 149 and placed into digital storage 150. 
A sticker sheet 118 and a first cut version of a photocoUage 
is generated 148 and sent to customer for review and 45 
modification. The sticker sheet 118, first cut version of the 
photocoUage and the layout pages 160 can be sent to the 
customer along with their prints. If modifications to the first 
cut photocoUage are desired the customer selects images 
from the sticker sheet 118 places them on the first cut 50 
photocoUage or on new page layout sheets or cover sheets 
160 as they wish 151. Tliey would also select the type of 
product 153 from a standard order form suppHed with 
sticker sheet 118. The customer then returns the modified 
first cut photocoUage, new cover and layout sheets 160 to the 55 
retailer 155 where the pages are scanned with the cover and 
layout sheet scanner 126 to create a photocoUage order. The 
photocoUage order is processed to determine which images 
are to be used in the photocoUage, the cover, the layout and 
the product type 157. The unique identifiers from the eo 
reduced resolution images are then associated with the 
images stored in the online storage facUity and these full 
scale images are retrieved 152. Tlie photocoUage is then 
constructed and printed 154 and sent to the customer 156. 

Depending upon the method employed for rendering the 65 
image identification number within the sticker sheet a vari- 
ety of image processing techniques would be employed. In 



the case of human readable image identification numbers 
optical character recognition software is employed that 
translates the scanned image of the characters to a digital 
number representation. In the case where the image identi- 
fication was hidden within the pixels of the reduced reso- 
lution image, a reverse of the data embedding signal pro- 
cessing technique is employed. The images are processed to 
remove any rotation and scale variations introduced in the 
printing and scanning steps. The result of this processing is 
to generate a list of the identification numbers of the images 
which the customer desires to have placed into the photo- 
coUage. 

The invention has been described with reference to a 
preferred embodiment However, it wiU be appreciated that 
variations and modifications can be effected by a person of 
ordinary skiU in the art without departing from the scope of 
the invention. 

PARTS LIST 

6 photographic prints 
8 camera media 

10 cartridges 

11 keepsakes 

12 memory cards 

14 processing facility 

15 CDs 
17 albums 
20 data store 

22 digital processing 
24 processing bag 
28 kiosk 
32 pixel data 
34 me ta -data 

36 order description 

37 goal generation 

38 processing goals 
40 products 

48 processing goals 

49 processing goals 
62 culUng 

64 grouping 
66 enhancing 
68 annotating 
70 associating 
72 composing 

118 sticker sheets 

119 numeric identifier on stickers 
142 exposed film 

144 film processed 

145 film digitization 

146 prints returned 

148 materials for photocoUages returned to customer 

149 unique ID 

150 digital storage 

151 image selection and placement 

152 high resolution image retrieval 

153 product decision 

154 photocoUage printing 

155 return of materials 
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156 finished product sent out 7. The method of claim 5 wherein the set of data was at 

157 layouts scanned least partially contained in the data base prior to the step of 
160 page layout sheet obtaining a digital version of a plurahty of images. 

162 cover layout sheet ^® method of claim 7 wherein the step of employing 
^ ^ ^ . , , - _ , , , ^ the parameters further comprises constructing the photocol- 

164 identifier tor page layout sheet or cover sheet , . j -.u «i. . c j * *i. * • * j ■ 

What is claimed is* accordance with the set of data that is stored in the 

^ . *i. J P * u . 11 data base relating to one of the following: pixel information; 

1. A method of constructmg a photocollage comprising i-r - r 

the ste s of ' o r- » metadata; order information or customer profile information. 

, . . * . - , 9. The method of claim 2 further comprising the step of 

obtammg a version of a plurality of miages; lO ^^^^^^^^ , ^^^y ^^^^^^ photocollage is accordance 

storing the digital images m a database; ^^^^ alteration made by the stickers, 

determining a set of parameters that pertain to the digital lo. The method of claim 2 further comprising the step of 

images such that each of the digital images contains at generating a final version of the photocollage is accordance 

least one of the parameters; ^jtjj ((jg ^ markup method using a writing instrument, 

sorting the images according to the parameters; 11. The method of claim 1 further comprising means for 

employing the parameters to automatically construct a of adding material to the photocollage. 

first cut version of the photocollage from the stored 12. The method of claim 11 wherein the means for adding 

images by generating a plurality of hardcopy pages of additional material further comprises means for adding 
stored images; 20 additional images to the photocollage that are stored in the 

returning the first cut version and a set of stickers to a data base. 

user; and 13. The method of claim 11 wherein the means for adding 

altering the pages with the set of stickers and each of the additional material further comprises means for adding 

stickers can be placed over one of the images so that a material selected from a predetermined set of graphics, 
revised version of the photocollage which replicates the 25 14, The method of claim 1 wherein the step of employing 

altered pages can be created. the parameters to automatically construct the photocollage 

2. The method of claim 1 wherein the step of altering the further comprises selecting a theme to construct the photo - 
pages further comprises each of the stickers having a thumb- collage . 

nail representation of one of the images containing a unique 15, The method of claim 1 wherein the step of employing 

identifier that cross references that image to the data base. the parameter by selecting of a theme further comprises 

3. The method of claim 1 wherein the photocollage which employing image understandmg principles to identify a key 
is constructed in the step of employing is a scaled rep re- characteristic in at least one of the image and apply the key 
sentation of the photocollage. characteristic throughout the database to generate the pho- 

4. The method of claim 1 wherein the step of providing tocoUage in accordance with the key characteristic, 
means for altering the pages further comprises providing ^5 16. The method of claim 1 wherein the photocollage is a 
means for using a writing instrument to indicate modifica- computer based photocollage contained in a digital file and 
tions to the images and the reconstruction of the photocol- the means for altering the pages further comprises software 
lage. means for modifying the images on pages. 

5. The method of claim 1 wherein the step of employing 17. The method of claim 16 wherein the software means 
the parameters further comprises constmcting the photocol- ^ is operative to display the photocollage and to replace 
lage using a set of data that is stored in the data base relating images on the pages with images selected from any image 
to one of the following: pixel information; metadata; order within the database. 

information or customer profile information. 18. The method of claim 16 wherein the software means 

6. The method of claim 1 wherein the step of employing is operative to alter the page layout with a different page 
the parameters further comprises constructing the photocol- template. 

lage employing one of the following techniques: culling; 

grouping; enhancing; annotating; composing; or associating. ***** 
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